A Natural Human-Computer Interface for Controlling Wheeled Robotic Vehicles

Robots are used increasingly to execute dangerous tasks and military missions. Autonomous robots are the warriors of the future, executing missions without requiring continuous supervision. Multimodal interfaces are the interfaces of the future in which speech, gestures, gaze, and other modalities are combined to provide a natural way for humans to communicate with machines. In this thesis I present a multimodal interface that was built to control and task wheeled robotic vehicles for military missions using spoken language, keyboard, mouse, touch, and gaze inputs, which can be used simultaneously. I developed a framework for multimodal command and control applications in which a novel fusion technique is used to combine these inputs. This framework was used to add multimodal interaction to Flatscape, a military mission planning and execution tool. With Flatscape and this new multimodal interface, natural interaction methods can be used to control robots directly or assign missions for them to execute autonomously.